ESD-TR-71-387 


>a 

oJ 


s 


ui 

-J 

a 


COMPUTER  PROGRAM  TESTING 


William  L.  Trainor,  Captain,  USAF 
R.  W.  Harris,  First  Lieutenant,  USAF 


November  1971 


TECHNICAL  REQUIREMENTS  AND  STANDARDS  OFFICE 
HQ  ELECTRONIC  SYSTEMS  DIVISION  (AFSC) 

L.  G.  Hanscom  Field,  Bedford,  Massachusetts  01730 


Approved  for  public  release; 
distribution  unlimited. 


LEGAL  NOTICE 

— 

When  U,  S.  Government  drawings,  specifications  or  other  data  are  used  for  any 
purpose  other  than  a  definitely  related  government  procurement  operation,  the 
government  thereby  incurs  no  responsibility  nor  any  obligation  whatsoever;  and 
the  fact  that  the  government  may  have  formulated,  furnished,  or  in  any  way  sup¬ 
plied  the  said  drawings,  specifications,  or  other  data  is  not  to  be  regarded  by 
implication  or  otherwise  as  in  any  manner  licensing  the  holder  or  any  other  person 
or  conveying  any  rights  or  permission  to  manufacture,  use,  or  sell  any  patented 
invention  that  may  in  any  way  be  related  thereto* 


OTHER  NOTICES 


Do  not  return  this  copy.  Retain  or  destroy. 


COMPUTER  PROGRAM  TESTING 


William  L.  Trainor,  Captain,  USAF 
R.  W.  Harris,  First  Lieutenant,  USAF 


November  1971 


TECHNICAL  REQUIREMENTS  AND  STANDARDS  OFFICE 
HQ  ELECTRONIC  SYSTEMS  DIVISION  (AFSC) 

L.  G.  Hanscom  Field,  Bedford,  Massachusetts  01730 


Approved  for  public  release; 
distribution  unlimited. 


FOREWORD 


This  Technical  Report  was  written  to  be  one  of  a  series  of  such 
reports  that  will  be  incorporated  into  a  Computer  Program  Management 
Handbook,  As  such,  this  report  is  directly  applicable  for  use  by 
Electronic  Systems  Division  system  program  managers  in  the  acquisi¬ 
tion  of  computer  program  configuration  items. 

Supplemental  guidance  concerning  computer  program  testing  may  be 
found  in  AFR  80-lU,  Test  and  Evaluation  of  Systems,  Subsystems,  and 
Equipment,  and  AFSC  Design  Handbook  DH  4-2,'  Electronic  Systems  Test 
and  Evaluation. 

Since  this  report  was  written,  Capt.  Trainor  has  been  reassigned 
to  the  Air  Force  Avionics  Laboratory,  Wright-Patterson  Air  Force  Base, 
Ohio. 

This  technical  report  has  been  reviewed  and  is  approved. 


CARMINE  PINTO,  Chief 
Tech  Rqmts  &  Stds  Office 
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ABSTRACT 


This  Technical  Report  addresses  the  general  area  of  computer  program 
testing.  The  test  program  for  computer  program  is,  in  many  canes, 
different  from  that  for  equipment  due  to%  the  uniqueness  of  the  computer 
program  configuration  item.  Thus,  testing  concepts  for  computer  pro- 
•  grams  are  not  always  widely  understood.  -This  Technical.  Report  attempts 

to  clarify  computer  program  testing  requirements  and  procedures  by  con¬ 
sidering  such  topics  as  test  requirenenti  documents  and  test  plans/ 

.  procedures/reports.  The  concepts  of  informal  versus  formal  testing 

are  introduced,  which  lead  to  the  subjects  of  preliminary  and  formal 
qualif icat Lon  testing.  Subprogram,  functional,  end  computer  program 
configuration  item  levels  of  testing  are  also  explained,  and  these 
ideas  are  in  turn  related  to  Informal  versus  formal  testing. 
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SECTION  I 


INTRODUCTION 


1.  General .  Testing  performed  as  an  integral  part  of  the  acquisition 
process  is  governed  by  AFR  80-l4  and  is  addressed  in  this  document  as 
Development  Test  and  Evaluation'  (DT&E) ,  and  Operational  Test  and  Evalua¬ 
tion  (OT&E).  The  purpose  of  the  total  tedt  effort  is  to  verify  the  perfor¬ 
mance  requirements  and  compliance  with  specifications  of  configuration 
items,  subsystems,  and  the  total,  integrated  system. 

2.  Development  Test  and  Evaluation.  The  DT&E  effort  is  divided  into 
the  two  areas  of  Cl/Subsystem  Test  and  System  Test.  The  functions  of 
each  of  the  two  areas  of  DT&E  are  given  below. 

a.  Cl/Subsystem  Test.  The  Cl/Subsystem  testing  effort  consists  of 
the  development  testing  and  evaluation  of  the  individual  configuration 
items  (CIs),  subsystems  and,  in  certain  cases,  the  complete  system.  The 
Air  Force  actively  participates  in,  evaluates,  and  controls  the  Cl/Sub- 
system  testing;  however,  the  tests  are  condUcted  predominantly  by  the 
contractor  who  is  under  Air  Force  Systems  Cbmmand  (AFSC)  direction  and 
control . 


(l)  The  overall  objective  of  the  Cl/Subsystem  effort  is  the 
qualification  of  all  CIs  and  subsystems  or  segments,  thereoy  preparing 
each  element  of  the  system  for  the  subsequent  System  test  program.  The 
total  objective  is  fulfilled  by  the  following  subordinate  accomplish¬ 
ments  : 


(a)  Engineering  test  and  evaluation  necessary  to  the 
development  of  an  acceptable  design. 

(b)  Preliminary  Qualification  Testing  (PQT)  to  confirm 
the  functional  integrity  of  mission  critical  functions . 

(c)  Formal  Qualification  Testing  (FQT)  of  each  Cl,  or 
group,  to  both  environment  and  functional  performance  requirements. 

(d)  Reliability  test  and  analysis  which  confirms  reli¬ 
ability  goals  and  defines  potential  problems. 

(e)  Integration  of  hardware,  computer  programs,  and 
personnel  subsystems . 

(f)  Qualification  of  system  segments  or  subsystems  as 
specified  in  performance  requirements . 

(2)  Government  control  cf  the  Cl/Subsystem  test  program  is 
established  primarily  through  provisions  of  the  contract  and  require¬ 
ments  of  thi  specifications.  Although  government  personn  1  will 


1 


normally  not  conduct  any  Cl/Subsystem  testing,  it  is  intended  that  the 
SFO  will  designate  government  representatives,  on  site,  to  perform  as 
monitors  to  determine  the  test  progress,  adherence  to  test  procedures, 
validity  of  collected  data,  and  performance  of  the  equipment  and  com¬ 
puter  programs.  Normally,  all  Computer  Program  Conf  igural  ion  Item 
(CPCl)  testing  will  be  conducted  is  an  integral  part  of  tie  Cl/Sub- 

system  effort.  • 

b.  System  Test .  The  System  testing  effort  consists  of  testing  and 
evaluation  spanning  the  integration  of  subsystems  into  a  c  omplete  system,  • 

and  development  tests  of  tne  completed  system  in  as  near  an  operational 
configuration  and  environment  as  practicable.  System  testing  is  an  Air 
Force  effort  with  contractor  participation,  under  AFSC  direction  and 
control,  and  with  active  operating  and  supporting  command  participation. 

Actual  test  operation  and  maintenance  should  be  performed  by  military 
personnel  who  have  received  formal  system  training. 

The  objective  of  System  testing  is  the  fomal  qualification 
of  the  system  specification  requirements.  Specific  objectives  of  the  Sys¬ 
tem  test  effort  are: 

(a)  Demonstrate  that  the  system  can  perfoim  the  mission 
as  specified  in  Section  3  of  the  System  Specification. 

(b)  Verify  results  of  Cl/Subsystem  testing  with  Air  Force 
crews  in  a  live  operational  environment. 

(c)  Qualify  and/or  demonstrate  the  performance  of  CIs 
which  require  full  system  operation  for  qualification. 

(d)  Verify  the  adequacy  and  compatibility  of  the  main¬ 
tenance  and  supply  support  concepts  as  developed. 

(e)  Determine  the  safety  characteristics  i  f  the  system, 
and  procedures  necessary  to  operate  and  maintain  the  system. 

(f)  Provide  sufficiently  trained  personnel  for  the  operat¬ 
ing  commands  to  assume  operational  evaluation  tasks  during  the  OT&E  phase. 

(g)  Verify  required  technical  handbooks  and  manuals. 

3 •  Operational  Test  and  Evaluation.  The  OT&E  testing  effort  normally 

follows  System  testing  and  completes  the  testing  phase  of  the  Program  • 

Management  Plan  (FWP).  OT&E  tests  will  be  conducted  by  t‘'.e  appropriate 

operating  command  with  technical  support  by  Air  Force  Sys  .ems  Command 

and  Air  Force  Logistics  Command. 
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SECTION  II 


TEST  DOCUMENTATION 


1.  Introduction .  There  are  several  types  of  test  documet nation  which, 
collectively,  form  the  basis  for  an  effective  test  program.  Broadly, 
these  can  be  classed  as : 

a.  Test  Requirements,  which  are  Section  4  of  each  System  (Type  A) 
or  Cl  (Type  3  or  C)  Specification. 

b.  Test  Plans,  which  are  usually  the  product  of  a  validation  phase 
and  should  reflect  the  overall  planning  for  test  and  evaluation  of  the 
system  or  a  subsystem. 

c.  Test  Procedures,  which  are  the  detailed  procedural  information 
for  conducting  each  test  delineated  in  the  test  plans  (ref.  "b"  above). 

d.  Test  Reports,  which  summarize  the  results  and  analysis  of  each 
test  conducted. 

2.  Test  Requirements  Documents.  Section  4,  "Quality  Assurance,"  of  each 
System  (Type  A)  or  Cl  (Type  B  or  C)  Specification  contains  specified  con¬ 
tractual  requirements  for  testing  of  the  respective  system  or  Cl.  This 
specification  section  should  depict  a  requirement  to  test  each  performance 
and  design  requirement  contained  in  Section  3  of  the  specification. 
Generally,  Section  4  of  a  A  specification  will  specify  requirements 
for  System  Test,  and  Section  4  of  a  Type  B  or  C  specification  will 
specify  requirements  for  Cl/Subsystem  Test. 

a.  Section  4,  System  Specification.  Section  4,  "Quality  Assurance," 
contains  the  requirements  for  the  8ystem  test  program.  These  require¬ 
ments  must  be  relatable  to  performance/design  requirements  stated  in 
Section  3*  Therefore,  Section  4  will  normally  be  limited  to  system 
level  test  requirements,  but  will  also  include  requirements  for  Cl/Sub- 
system  engineering  tests,  qualifications,  and  reliability  tests  which 

can  be  accomplished  only  at  the  system  test  locatlon(s) .  The  principal 
content  of  Section  4,  however,  is  the  specification  of  System  test 
requirements . 

b .  Section  4,  CPCI  Part  I  Specification.  Test  requirements  are 
developed  initially  by  the  contractor  for  incorporation  into  Section  4 
(Quality  Assurance)  of  the  Part  I  specification.  This  section  should 
identify  test  methods  (to  the  level  of  detail  necessary  to  clearly 
establish  the  scope  and  accuracy  of  the  methods)  to  be  employed  in 
qualifying  the  CPCI  against  all  performance  and  design  requirements 
specified  in  Section  3  of  that  Cl  specification.  In  addition,  it  should 
identify  requirements  for  government- furnished  equipment  and  facilities 
to  support  the  contractor's  computer  program  test  and  evaluation,  as  well 
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as  designate  those  performance  characteristics  to  be  demonstrated  and/or 
verified  during  preliminary  qualification  tests  and  demonstrations, 
formal  qualification  testing,  and  System  testing. 

(1)  Requirements  of  Section  4  should  be  specified  to  .he  level 
of  detail  which: 

t 

(a)  Designates  verification  requirements  and  methods  for 
each  performance/design  requirement  identified  in  Section  3 •  The  method 

of  verification  to  be  specified  may  include  inspection,  review  of  analyt-  ♦ 

ical  data,  demonstration  tests,  and  review  of  test  data. 

(b)  Clearly  establishes  the  scope  and  accuracy  of  the  test 

method . 

(2)  Types  of  Cl/Subsy3test  tests  which  may  be  specified  in 
Section  4  of  a  CPCI  Specification  are: 

(a)  Computer  Programming  Test  and  Evaluation .  which  are 
tests  conducted  primarily  to  support  the  design  and  development  process. 

They  are  listed  in  the  specification  only  when  they  meet  one  of  the 
following  criteria: 


1.  They  are  intended  to  be  the  only  source  of  data  to 
qualify  specific  requirements  in  Section  3* 

2.  They  must  be  accomplished  as  part  oi  an  integrated 
test  program  involving  other  systems/equipment/programs . 

3»  They  require  the  use  of  government-! ornished  test 
facilities  or  equipment . 

(b)  Preliminary  Qualification  Tests,  which  are  formal  tests 
oriented  primarily  towards  verifying  portions  of  the  CPCI  prior  to  formal 
qualification  tests  of  the  integrated  CPCI. 

(c)  Formal  Qualification  Tests,  which  are  formal  tests 
oriented  primarily  towards  testing  of  the  integrated  CPCI,  using  oper¬ 
ationally  configured  equipment . 

(3)  The  System  test  requirements  to  be  identified  in  Section  4 
concern  those  performance/design  requirements  which  cannot  be  verified 
until  System  testing.  Emphasis  should  be  placed  on  minimizing  these 
types  of  requirements  and,  if  possible,  eliminating  them  altogether. 

This  will  ensure  that  the  CPCI  that  is  Cl/Subsystem  tested  has  in  fact 
attained  a  high  degree  of  confidence.  Also,  this  will  allow  the  System 
test  program  to  proceed  with  its  prime  objective,  testing  of  system 
level  requirements . 

(4)  An  important  point  to  remember  is  the  differentiation 
between  Section  4  and  other  testing  documents .  Section  4 ,  by  virtue  of 
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being  part  of  the  Part  I  CFCI  Specification,  is  a  contractual  document, 
and  the  contractor  is  required  to  conduct  only  those  tests  tnat  are 
called  out  in  that  section.  Any  changes  or  deviations  from  these  re¬ 
quirements  (once  baselined)  must  be  approved  by  the  System  Program 
Office  (SPO)  through  Engineering  Change  Proposal  (ECP)  actions.  Thus, 
careful  attention  must  be  given  to  ensuring  that  the  testing  has  been 
scoped  properly,  and  the  test  requirements  are  complete  and  acceptable. 

3*  Test  Plans/Procedures/Reports .  Figure  1  shows  a  typi  .-a!  test  docu- 
mentation  'tree”  for  a  large  system.  The  tree  is  constructed  from  the 
list  of  approved  test  plan/procedure/report  data  item  descriptions  (DIDs) 
contained  in  AFSCM/AFLCM  310-1,  Vol  II. 

a.  System  Test  Plan.  The  System  Test  Plan,  DID  T-101-1,  Is  the 
"top  document"  for  the  test  program  which  structures  and  unifies  all 
subsequent  test  plans.  Its  purpose  Is  to  provide  an  over  ill  outline 
of  the  total  system  test  program  to  include  planning  factors,  objectives, 
and  scope  of  all  phases  of  the  test  program.  The  system  -'ontractor  pre¬ 
pares  the  System  Test  Plan  (DID  T-101-1),  normally  for  de  Ivery  in  the 
Conceptual  or  early  Validation  Phase .  This  plan  will  con  ain  basic  test 
planning  information  for  all  phases  of  the  test  program  and  cover  the 
life  cycle  of  the  system  through  the  end  of  the  Full  Seal.*  Development 
Phase.  The  scope  and  level  of  detail  of  the  Information  is  sufficiently 
broad  and  comprehensive  to  provide  basic  test  inputs  to  tee  Program 
Management  Plan  (FMP)  and  sufficiently  detailed  to  provide  the  basis 
for  preparing  all  subsequent  test  plans- -particularly  Cl/bubsystem 
(DID  T-102-1  and  DTD  T-103-l)  and  System  (DID  T- 106-2)  plans. 

(l)  The  System  Teat  Plan  (DID  T-101-1)  is  based  upon  the  test 
concepts  and  requirements  contained  in  the  Program  Manageiient  Plan  (FMP), 
the  System  Specification,  find  related  system  engineering  documentation. 

It  covers  all  aspects  of  the  system,  including  hardware,  facilities, 
computer  programs,  personnel,  and  procedural  data  with  re-pect  to  such 
consideration  as  the  following: 

(a)  Organizational  responsibilities  for  testing. 

(b)  Basic  system  test  concepts  and  objectives  for  Cl/Sub- 
system,  System,  Implementation,  and  Acceptance  tests. 

(c)  Overall  system  test  operations,  including  te3t  control 
requirements  and  test  support  requirements . 

(d)  Test  evaluation  requirements. 

(e)  Test  reporting  requirements 

(f)  Overall  test  schedules. 
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(2)  A  primary  function  of  the  System  Test  Plan  (DID  T-101-1 ) 
is  to  guide  the  contractor's  planning,  analysis,  and  system  engineer¬ 
ing  activities  related  to  the  test  areas.  During  the  Validation  Phase 
the  contractors  expand  basic  planning  and  guidance  information  contained, 
in  the  System  Test  Plan  (DID  T-101-l)  and,  in  effect,  replace  it  with 
the  f ollowing  documents : 

(a)  Cl/Subsystem  Test  Plan  (DID  T- 102-1  and  T-K>3-l). 

(b)  Inputs  to  the  System  Test  Flan  (DID  T-10fc-2) . 

(c)  Inputs  to  the  Implementation  Test  Plan  (multisite 
systems;  DID  T-»llU-l) . 

(3)  Special  problems  to  be  considered  concerning  cr uputer  pro¬ 
grams  may  include  provisions  for  computing  equipment  required  for  con  ■ 
tractor  development  and  testing  of  computer  programs.  The  CI-Subs>mten 
test  cycle  for  computer  program  CIs  will  normally  begin  early  during 
the  Full  Scale  Development  Phase;  It  may  have  to  be  Initiated  using  a 
prototype  computer  and  peripheral  equipment,  or  in  same  canes  by  simu 
lation  on  existing  computers.  As  development  of  the  CFCI  progresses, 
testing  will  encompass  progressively  expanded  groups  of  routines,  and 
will  require  additional  Items  of  computing  equipment  for  realistic  test¬ 
ing  of  functions.  Preliminary  qualification  tests/ demons tration  wilJ 
normally  occur  at  the  contractor's  plant,  or  other  locations  of  available 
equipment.  Formal  qualification  of  the  computer  program  CT  may  require 
scheduled  time  at  the  System  test  site  for  testing  those  computer  pro 
gram  functions  that  depend  upon  the  operationally  configured  system/ 
equipment  end  for  which  simulation  is  not  adequate .  For  a  multisite 
system  which  requires  adaptation  of  the  operational  computer  program 

Cl  at  each  site  location,  the  expected  sequence  of  events  at  each  fol¬ 
low-on  site  will  be  installation  and  checkout  of  equipment  at  the  site 
facility,  installation  testing,  adaptation,  installation  and  checkout 
of  computer  programs,  and  implementation  testing  (See  Figure  2). 

b.  Cl/Subsystem  Test  Plan/Procedures/Reports  (Equipment).  The  Cl/ 
Subsystem  Test  Plan/Procedures  data  item,  DID  T-102-1,  is  the  overall 
planning  document  for  the  Cl/Subsystem  test  program.  It  is  subordinate 
(in  the  reft  planning  tree)  only  bo  the  System  Test  Plan  (DID  T-101-1 ) . 

This  document  should  provide  ccjraplete  planning  information  on  each  Cl/ 
Subsystem  test  specified  in  each  of  the  Cl  specifications  on  contract- - 
i.e.,  System  Specification,  Cl  Specification,  Critical  Component  Spec:  - 
flcation,  Military  Specification,  and  other  contractual  documents  that 
include  Cl/Subsystem  test  methods  and  success  criteria.  Note  that,  test 
planning  and  procedures  information  for  Cl/Subsystem  testing  of  computer 
programs  should  not  be  included  In  this  document.  In  the  case  where  the 
system  involves  computer  programs,  an  additional  test  data  item  is  used 
to  detail  the  planning  and  procedural  information  for  testing  of  the 
computer  programs.  This  document  is  Cl/Subsystem  Test  Plan/Procedures 
(Computer  .rogram),  DID  T-103-1,  which  shpuld  he  referenced  in  DID  T-102-1. 
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This  data  item  is  normally  obtained  in  the  Validation  Phase  as  a  com¬ 
plete  plan  and  individual  test  information  sheets  are  then  updated  in 
the  Full  Scale  Development  phase  prior  to  each  test .  Test  reports  are 
obtained  under  DID  T-118-1. 

c .  Cl/Subsystem  Test  Plans/Procedures/Report  (Computer  Programs). 

The  computer  program  C I/Subsystem  Test  Plan /Procedures,  DTD  T-103-1,  is  • 

a  subordinate  and  supplementary  document  to  the  overall  Cl/Subsystem 
Test  Plan/Procedure,  DID  T-102-1.  When  the  system  involves  computer 
programs  and  equipments,  DID  T-103-1  should  be  placed  on  contract  « 

along  with  DID  T-102-1;  DID  T-102-1  will  apply  to  the  Cl/Subsystem  test¬ 
ing  of  all  equipments  and  reference  DID  T-103-1  which  will  cover  the  Cl/ 

Subsystem  testing  for  the  computer  programs. 

(l)  The  Cl/Subsystem  Test  Plan  for  computer  programs  is  a 
contractor-prepared  document  which  establishes  criteria,  general  methods, 
responsibilities,  and  overall  planning  for  Cl/Subsystem  testing  of  a 
CPCI.  Normally,  the  test  planning  information  is  obtained  in  the  Vali¬ 
dation  Phase  as  a  complete  plan  applicable  to  all  computer  programs  of 
the  system.  Generally,  only  one  plan  (and  one  volume)  will  be  prepared 
for  a  single  system  or  contract- -this  allows  for  the  presentation  of  an 
integrated  test  plan  which  will  apply  to  all  computer  programs  for  a 
particular  system.  Exception  to  this  guidance  can  be  obtained  for  the 
individual  system.  Sections  of  the  plan  which  apply  to  computer  pro¬ 
grams  for  subsystems  for  which  inadequate  information  exists  at  time  of 
writing  of  fhe  plan,  can  be  updated  at  a  later  date.  When  this  is  the 
case,  the  section  should  be  included  nonetheless  with  the  remark,  "to 
be  completed  later." 

(a)  The  plan  should  contain  detailed  information  concern¬ 
ing  the  implementation  of  preliminary  and  formal  qualification  tests, 
along  such  lines  as  the  following': 

1.  Locations  at  which  the  tests  will  be  conducted, 
and  schedules  relative  to  milestones  in  the  overall  acquisition  schedule. 

2.  General  methods  for  preparation  of  input  data — 
i.e.,  simulation  and/or  generation  vehicles  to  be  used. 

General  procedures  for  test  conduct,  and  respon-  . 

sibilities  for  test  direction,  operation,  and  observation. 

4.  General  procedures  for  analysis  of  test  results. 

5-  Requirements  for  other  computer  programs,  equip¬ 
ment,  and  facilities. 

6.  Personnel  requirements,  including  numbers,  re¬ 
sponsibilities,  and  particular  knowledge  and  skills  required. 
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(b)  The  plan  may  alao  set  forth  the  requirements  and 
procedures  for  controlling  and  documenting  uhe  Cl/Subsystem  test 
program,  including  procedures  for  preparing,  reviewing  and  revising 
documentation  of  specific  test  procedures;  requirements  and  pro¬ 
cedures  for  preparing  and  reviewing  reports  of  individual  quuli.fi- 

♦  cation  tests;  summaries  of  the  Cl/Subsystem  test  program  or  phases 
thereof;  and  other  reports  related  to  the  Cl/Subsystem  test  activity. 

♦  (2)  The  Cl/Subsystem  Test  Procedures  are  produced  by  the  con¬ 
tractor  during  the  Full  Scale  Development  Phase . 

(a)  For  each  individual  Cl/Subsystem  Qualification  Test 
(PQT  or  PQT)  for  a  CPCI,  the  contractor  will  prepare  a  Cl/Subsystem 
Test  Procedure  in  accordance  with  Section  2  of  DID  T-103-1.  These 
procedures  are  prepared  Incrementally  during  the  Full  Scale  Development 
Phase  and  submitted  prior  to  the  test  date  of  the  PQT  or  FQT  to  which 
they  apply.  Information  included  in  a  test  procedure  includes  the 
f  ollowing : 


1.  Location  and  schedule  of  the  test,  briefings,  de¬ 
briefings,  and  any  associated  data  reduction/analysis . 

2.  References  to  applicable  test  plan,  specifications, 
manuals  and  handbooks . 

Detailed  objectives  of  the  test. 

b .  Requirements  and  responsibilities  for  console 
operators,  est  directors,  technical  consultants,  data  analysts,  or 
other  essential  test  personnel. 

j>.  Requirements  for  other  computer  programs  (other 
than  the  CPCI  being  tested)  or  equipment. 

b.  Test  operating  procedures  to  specify  how  to 
initiate  the  computer  program  operation,  maintain  the  computer  program 
operation,  and  terminate  and/or  restart  the  computer  program  operation. 

A  detailed  test  description  for  each  te3t  (or 
portion  of  a  test)  to  be  performed.  This  description  should  include 
detailed  information  on  test  inputs,  outputs,  events,  expected  results, 
reactions  to  be  verified,  and  methods  of  verification. 

8.  Requirements  and  procedures  for  recording,  re¬ 
auction  and  analysis  of  test  data. 

(3)  The  Cl/Subsystem  Test  Reports  for  computer  program  tests 
ore  obtained  under  DID  T-118-1.  These  are  contractor  prepared  docu¬ 
ments,  compiled  incrementally  during  the  Full  Scale  Development  Phase. 
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Generally,  one  test  report  (in  accordance  with  DID  T-ll8-l)  is  prepared 
for  each  PQT  or  FQT  conducted. 

d.  System  Test  Plan/ Procedure/Report .  DID  T-106-2  Is  the  basic 
document  which  provides  the  overall  integrated  outline  of  the  System 
Test  Program.  It  includes  all  planning  factors,  scope,  detailed  test 

objectives,  identification  of  test  areas,  responsibilities  of  partici-  # 

pating  agencies,  and  associated  information  necessary  to  implement  the 
minimum  acceptable  performance  requirements  of  the  Program  Management 
Directive  {PMD)  and/or  the  System  Specification.  Normally,  the  con-  « 

tractor  will  prepare  the  System  Test  Plan  and  Procedures  (DID  T-106-2) 
with  the  intent  being  to  establish  a  test  and  evaluation  program  to 
ensure  that  the  sy stem/ equipment /computer  programs  meet  the  minimum 
acceptable  performance  requirements  of  the  PMD  and  System/CI  Specifi¬ 
cations  in  as  realistic  and  complete  an  operational  environment  as 
practicable  Test  Reports  are  obtained  under  DID  T-120-2. 

e.  Installation  Testing.  Installation  Testing,  normally  conducted 
after  completion  of  each  installation,  Includes  Preshakedown  Tests, 
which  are  performed  as  a  combination  alignment  check  and  test  to  assure 
that  the  installation  is  properly  completed;  Shakedown  Tests,  which  are 
performed  to  assure  that  all  detected  marginal  parts  and  material  have 
been  eliminated  and  that  the  installation  is  ready  for  operational 
tests;  and  Operational  Tests,  which  are  performed  to  demonstrate  that 
the  equipment  i3  properly  installed  and  is  capable  of  performing  its 
operational  mission  up  to  a  specified  interface  with  other  portions  of 
the  subsystem/ system. 

(1)  The  Installation  and  Checkout  Plan  (I&C),  DID  T-112-1,  is 
initially  prepared  by  the  contractor  during  the  Validation  Phase  and 
later  exparded  to  reflect  the  system  engineering  and  detail  design 
activity  of  the  Full  Scale  Development  Phase .  Procedures  are  prepared 
to  implement  the  plan,  based  upon  assembly  levels  of  components  and 
consideration  of  all  interfaces.  In  general,  individual  items  are  in¬ 
stalled,  physically  and  functionally  checked  out,  then  ms’.ed  to  other 
subsystems .  This  graduated  process  is  applied  successively  until  the 
complete  system  is  ready  for  system  tests . 

(2)  I&C  plans  are  reviewed  and  approved  by  the  SPO  and  the 

procedures  selectively  reviewed  and  approved.  • 

(3:  Adaptation,  installation,  and  checkout  (AI&C)  of  the  com¬ 
puter  program  will  normally  occur  after  installation  and  checkout  of 
equipment  and  facilities .  Normally,  the  work  will  be  accomplished  by 
the  contractor  in  accordance  with  the  approved  plans  and  procedures 
obtained  under  DID  T-112-1.  At  the  System  Test  Site,  AI&C  of  the  com¬ 
puter  prog -am  may  be  followed  by  Formal  Qualification  Testing  prior  to 
initiation  of  the  System  Test.  Based  upon  the  experience  gained  at  the 
System  Test  Site,  the  AI&C  plan  may  be  modified  and  updated  for  use  at 
each  follow-on  site.  AI&C  of  computer  programs  at  follow-on  sites  will 
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include  the  same  basic  activities  as  for  AI&C  at  the  System  test  site 
and  will  occur  in  the  same  general,  sequence,  i.e.,  following  I&C  of  the 
equipinent/facilities,  but  prior  to  the  start  of  Implementation  Tests 
(see  Figure  2).  For  computer  programs,  appendices  may  be  used  to  cover 
unique  adaptation  features  for  each  site. 

(a)  The  computer  program  contractor  must  identify  require¬ 
ments  for  AI&C  of  the  computer  program  at  the  System  test  site,  and  for 
multisite  systems,  must  also  include  requirements  for  AI&C  at  subsequent 
site  installations.  Documentation  should  include  a  detailed  description 
of  all  aspects  of  the  adaptation,  installation  and  checkout  activities. 
Detailed  schedules  should  be  provided  and  all  support  requirements 
clearly  identified,  including  required  computer  time,  requirements  for 
other  system  equipments  such  as  communication,  display  consoles,  etc., 
and  requirements  for  trained  personnel.  Detailed  information  should 
also  be  provided  concerning  the  training  of  personnel  for  the  field 
locations,  scheduled  movement  of  field  personnel,  local  transportation, 
office  space  and  facilities,  and  living  quarters  at  remotely  located 
sites.  Procedures  should  be  prepared  indicating  in  detail  the  exact 
manner  in  which  the  plan  will  be  implemented. 

NOTE:  "Adaptation"  refers  to  the  process  of  inserting  into  the  computer 
program  the  coded  data  which  are  appropriate  to  the  geography  or  other 
char ac ter i s tics  of  the  given  site.  "Installation"  refers  to  insertion 
of  the  codec  instruction/data  into  the  computing  equipmen  ;  in  contrast 
with  equipment  installation,  computer  program  installation  typically 
involves  relatively  insignificant  effort  or  time.  The  major  task  of 
AI&C  is  the  checkout  activity,  which  may  in  some  cases  in  olve  assembly 
of  new  elements,  compiling,  and  extensive  debugging,  as  w*ll  as  person¬ 
nel  training  and  preliminary  rehearsals  of  FQT  procedures . 

f .  Implementation  Testing.  Implementation  Testing  is  e.  concept  which 
has  been  defined  to  meet  the  specific  requirements  of  muluibite  (i.e., 
ground  installation,  airborne  vehicles,  space  platforms,  or  tactical 
emplacements)  electronic  systems.  The  concept  is  based  on  experience 
which  has  shown  that  additional  testing  beyond  Installation  Testing  may 
be  required  at  successive  operating  sites  to  insure  that  these  sites, 
individually  and  collectively,  function  as  parts  of  a  system  in  accor¬ 
dance  with  the  requirements  of  the  system  performance  specification. 

These  tests  encompass  real-time  functional  testing  perfor.aed  at  the 
system  level  with  the  purpose  of  exposing  faults  and  providing  a.  demon¬ 
stration  to  the  user  that  the  installed  site  is  ready'  for  operational 
use . 


(l)  The  relation  of  the  System  Testing  and  Implementation  Test¬ 
ing  should  be  one  of  decreasing  complexity  and  sophistication.  To  the 
extent  that  system  performance  requirements  set  forth  in  the  system 
specification  have  been  demonstrated  by  System  testing,  a  complete  re¬ 
petition  of  the  effort  for  each  follow-on  system  is  not  warranted. 
Rather,  spt  clfic  functional  performance  measures  which  adequately  define 
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Figure  2.  Computer  Program  Adaptation,  Installation,  and  Checkout  for  a  Multi-site  System 


system  response  are  identified  at  an  early  point  in  time,  and  the  actual 
values  for  such  performance  measures  (figures  of  merit)  are  determined 
during  System  testing.  These  figures  of  merit  are  used  during  Inrole- 
mentation  Testing  as  the  criteria  by  which  to  gauge  functional  adequacy 
of  the  system.  In  general,  Implementation  Te6t^  should  not  attempt  to 
duplicate  the  comprehensive  testing  accomplished  during  System  testing. 

(2)  To  insure  operational  validity,  the  user  as  well  as  the 
system  designer  must  participate  in  Implementation  Test  plai  fling.  The 
user  is  interested  in  a  test  that  will  uncover  technical  incompatibilities 
or  functional  and  operational  deficiencies,  Thus,  the  manner  in  which 
the  system  is  to  be  employed  is  a  vital  factor  in  test  planning 

(3)  The  Implementation  Test  Plan  end  Procedures  arc  contractor- 
prepared  documents  obtained  under  DID  T-114-1.  The  Implementation  Test 
Plan  defines  the  overall  scope  of  the  implementation  tests,  the  objec¬ 
tives,  methods,  and  support  requirements  for  the  conduct  of  this  phase 
of  testing.  The  Implementation  Test  Procedures  (one  set  of  procedures 
for  each  test  conducted)  outline  a  step-by-step  set  of  instructions  and 
criteria  for  each  test  specified  in  the  plan.  Test  results  are  reported 
under  DID  T- 119-1 • 

(4)  The  computer  program  contractor(s)  prepare  inputs  to  the 
Implementation  Test  Plan/Procedures  much  the  same  as  for  the  System 
Test  Plan/Procedurea .  Since  implementation  testing  is  at  the  system 
level,  emphasis  will  be  placed  on  those  aspects  of  the  computer  program 
performance  which  depend  upon  complex  interactions  with  other  CPCIs, 
personnel,  equipment,  and  data  links  during  operation  of  the  entire 
system . 


(a)  Special  emphasis  will  be  given  to  those  portions  of 
the  CPCIs  (generally  the  operational  CPCl)  which  vary  from  site-to-site, 
i.e.,  environmental  data  values,  special  adaptation  parameters,  unlque- 
to-site  interfaces,  and  site-peculiar  program  routines. 

(b)  Inputs  to  the  Implementation  Test  Flan  relative  to 
computer  programs  should  be  similar  to  the  inputs  to  the  System  Test 
Plan  (DID  T- 106-2) .  In  general,  they  should  emphasize  those  aspect g  of 
system  performance  which  are  closely  associated  with  or  dependent  upon 
performance  of  the  CFCl(s).  Frequently,  the  inputs  may  represent  a 
selected  subset  of  the  System  Test  Plan  (DID  T- 106-2)  inputs. 

4.  Section  4,  CFCI  Part  II  Specification.  Normally,  Section  4  of  a  Cl 
Part  II  Specification  will  contain  quality  assurance  provisions  and 
acceptance  test  requirements  for  follow-on  production  items.  However, 
unlike  equipment  items,  there  is  no  follow-on  production  process  for  a 
computer  program  item;  thus,  the  concept  of  acceptance  tests  for  pro¬ 
duction  items  will  not  apply  to  a  CPCI.  Instead,  Section  4  of  a  CFCI 
Part  II  Specification  should  contain  two  subsections : 
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a.  1'es't-  Plan/Procedure  Cross  Reference  Index.  This  subsection 
should  contain  a  cross-reference  diagram  depicting  each  function  (a3 
delineated  in  the  CFCI  Part  I  Specification)  and  relating  these  func¬ 
tions  to  the  corresponding  test  plan/procedures  that  were  used  to 
qualify  the  individual  requirements  of  the  CFCI. 

b.  Other  Quality  Assurance  Provisions .  This  subsection  r  v.ould 
reference  and/or  specify  the  test/verification  requirements,  methods, 
and  procedures  which  apply  to  preparation  and  duplication  of  the  com¬ 
puter  program  (i.e.,  tapes,  card  decks,  etc.). 


SECTION  III 


COMPUTER  PROGRAM  Cl/SUBSYSTEM  TESTING 


1.  Introduction .  The  Cl/Subsystem  testing  of  CPCIs  serves  a  fourfold 
purpose : 

a.  To  establish  each  CFCI  as  a  qualified  end  Item  suitable  for  entry 
into  the  System  Test  Program.  This  qualification  is  accomplished  by 
verifying  the  performance  and  design  requirements  of  the  CFCI  Part  I 
Specification. 

b.  To  furnish  the  procuring  agency  with  the  proper  visibility  re¬ 
quired  for  effective  management  of  the  system.  This  is  accomplished 
through  Judicious  scheduling  of  PQTs,  thus  establishing  milestones  which 
will  provide  insight  into  the  progress  of  the  CPCI  design  and  develop¬ 
ment.  If  problems  are  encountered  in  the  computer  program  area,  early 
detection  can  be  made  and  more  effective  management  and  engineering  can 
be  focused  on  these  potential  problem  areas. 

c .  To  serve  as  a  standard  or  " straw  man”  about  which  the  contractor 
can  develop  his  internal  verification  procedures. 

d.  To  develop  a  comprehensive  test  (called  FQT)  which  can  be  utilized 
after  FCA  as  an  ongoing  test  tool/procedure  for  retesting  the  CPCI.  FQT 
is  not  only  used  for  formal  verification  prior  to  FCA  but  will  be  con¬ 
tinuously  updated  and  used  to  retest  the  CPCI  whenever  changes  are  made 
to  the  program. 

2.  Types  of  Cl/Subsystem  CPCI  Testing.  Cl/Subsystem  testing  of  CFCIs 
can  be  broadly  divided  into  two  types — informal  testing  and  formal  test¬ 
ing.  The  basic  difference  between  there  types  stems  from  the  documen¬ 
tation  requirements;  informal  testing  utilizer  the  contractor's  interned, 
test  documentation,  controls  and  procedures;  formal  testing  is  conducted 
in  accordance  with  Air  Force  approved  test  plains  and  procedures . 

a.  Informal  Testing.  Cl/flubsyrtrm  informal  testing  (referred  to  as 
Computer  Program  Test  and  Evaluation,  CFTW5)  will  usually  form  the  bulk 
of  the  contractor's  Cl/Subsyetem  test  effort.  It  is  designed  to  be  the 
contractor's  "in-house"  testing  and  requires  no  government  approved  test 
plans/procedures.  Generally,  the  entire  informal  test  effort  will  be 
documented  internally  by  the  contractor,  and  such  information  will  be 
made  available  to  the  procuring  agency  only  cn  demand.  Informal  testing 
begins  when  the  first  subprogram  is  coded  end  contimieo  throughout  the 
Full  Scale  Development  Haase. 

(l)  Informal  testing  la  required  cn  three  levels  for  a  step-by- 
step  validation  of  the  CPCI: 
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(a)  Subprogram  level  (parameter  teste) 

(b)  Functional  level  (assembly  teste) 

(c)  CPC I  level  (assembly  tests) 

(2)  Parameter  teiata  are  tests  of  the  information  processing 
logic  of  the  individual  CPCs.  Sets  of  inputs  are  prepared  which  cause 
all  the  logic  paths  to  be  exercised  The  resultant  outputs  are  checked 
against  hend  calculations  which  show  whether  the  CPC  has  been  coded 
correctly . 

(3)  Assembly  tests  are  testa  of  groups  of  CPCs  which  perform 
a  function  of  the  CPC  I,  or  in  later  stagt'3,  many  or  all  of  the  func¬ 
tions  of  the  CFCI.  Interfaces  between  the  CPCs  as  well  as  overall 
information  processing  are  tested.  Simulated  inputs  to  the  CPC I  are 
used,  and  the  CFCI  is  operated  in  real-time  or  simulated  real-time 
mode. 


(4)  The  informal  test  activities  are  conducted  solely  for  the 
purpose  of  providing  information  required  for  the  developmental  process. 
As  such,  they  do  not  require  recognition  by  the  procuring  agency  and 
would  not  normally  be  identified  in  the  Cl/Sub system  Test  Plan.  The 
planning  and  conduct  of  informal  testing  is  carried  out  in  accordance 
with  the  contractor's  internal  management  procedures,  and  within  the 
time  constraints  imposed  by  formally  scheduled  reviews  and  qualification 
testing.  Each  CPC  or  subprogram  must  pees  through  a  series  of  stages 
and  iterations,  consisting  of  such  operations  as  desk  checking,  elimi¬ 
nation  of  illegal  instructions,  parameter  testa,  functional  testing  with 
controlled  data  inputs,  assembly  with  additional  components  of  the  CPCI, 
assembly  testing,  performance  testing  with  simulated  Inputs  and,  finally, 
performance  testing  in  the  system  under  conditions  of  live  operations. 

(5)  The  Cl/Subsystem  Test  Plan  will  not  generally  Include  plan¬ 
ning  information  for  contractor  testing  which  is  conducted  as  an  integral 
part  of  the  design  and  development  process.  However,  the  plan  will 
typically  detail  the  contractor's  requirements  for  government  facilities 
and  other  support  required  for  conducting  the  tests .  These  will  include 
requirements  for  computing  and  peripheral  equipment  which  must  be  fur¬ 
nished  or  otherwise  made  available  for  the  contractor's  use  on  an  ap¬ 
propriate  schedule  during  the  Full  Scale  Development  Phase.  Where 
formal  qualification  of  the  CPCI  is  scheduled  to  occur  at  the  System 
test  site,  it  will  normally  be  preceded  by  a  phase  of  contractor  toot 
and  evaluation  associated  with,  adaptation,  installation,  and  checkout 

of  the  CFCI(s)  at  the  site.  The  Cl/Subsyetem  Test  Plan  should  include 
(or  reference  other  planning  documents  which  include)  requirements  for 
scheduled  use  of  the  facility,  system,  equipment,  other  computer  program 
CIs,  personnel,  and  other  items  needed  to  support  the  conduct  of  theae 
contractor  testa . 
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b.  Formal  Testing.  Cl/Subsystem  formal  testing  is  that  portion  of 
Cl/Subsystem  testing  which  is  conducted  in  accordance  with  Air  Force 
approved  test  plans/procedures.  Thus,  the  Air  Force  has  explicit  con¬ 
trol  of  the  type,  number,  and  frequency  of  tests  to  be  performed. 

(l)  The  Cl/Subsystem  formal  test  effort  is  divided  into  two 
distinct  types  of  testing- -Preliminary  Qualification  Testing  (PQT)  and 
Formal  Qualification  Testing  (FQT) .  PQT  is  designed  to  be  an  incre¬ 
mental  process  which  will  provide  the  procuring  agency  proper  visibility 
and  control  of  the  computer  program  development  during  the  time  period 
between  the  Critical  Design  Review  (CER)  and  FQT.  FQT  is  designed  to  be 
a  complete  and  comprehensive  test  of  the  CPCI  in  a  "one  step"  fashion 
Just  prior  to  FCA. 

(a)  Preliminary  Qualification  Testing  (PQT)  is  composed 
of  function  level  tests  (assembly  tests)  which  are  conducted  in  accor¬ 
dance  with  Air  Force  approved  test  plans/procedures  procured  under 
DED  T-103-1.  PQT  is  to  be  an  incremental  process  which  occurs  between 
CDR  and  FQT  of  a  CPCI's  development  process.  For  each  function  (of  the 
Part  I  CPCI  Specification)  which  is  designated  for  testing  during  PQT, 
a  separate  test  procedure  is  written  and  a  formal  test  conducted.  Note 
that,  generally,  not  all  functions  of  a  CPCI  are  tested  during  FQT  since 
experience  has  proven  that  it  is  both  too  costly  and  too  time-consuming. 
Instead,  omy  designated  functions  of  the  CPCI  are  tested  during  PQT; 
thus,  the  problem  becomes  one  of  the  selection  of  those  functions  which 
should  undergo  PQT.  Selection  of  these  functions  should  be  based  on 
the  following: 


1.  A  FQT  should  be  conducted  for  each  function  that 
is  "time- critical"  to  the  development  of  the  CPCI,  subsystem,  or  system. 
For  example,  the  compiler  function  of  a  utility  CPCI  may  be  designated 
"time-critical”  if  the  development  of  the  remaining  computer  programs 
depend  on  the  timely  development  of  a  compiler  to  be  used  for  compiling 
the  other  programs.  The  development  of  the  executive  function  of  an 
operational  CPCI  may  also  be  designated  "time-critical"  since  the 
orderly  progression  from  parameter  testing  to  assembly  testing  of  the 
operational  CPCI  would  require  the  timely  development  of  the  executive 
function . 


2.  A  PQT  should  be  conducted  for  each  function  that 
is  "performance-critical"  to  the  development  of  the  CPCI,  subsystem,  or 
system.  For  example,  the  executive  function  could  be  designated  "per¬ 
formance-critical"  as  well  as  "time-critical"  due  to  the  utilization  of 
various  exotic  scheduling  techniques.  As  another  example,  the  tracking 
function  of  a  CPCI  for  an  on-line  radar  system  might  be  designated  "per¬ 
formance-critical"  due  t,o  the  utilization  of  new  tracking,  smoothing, 
and  filtering  techniques . 

(b)  Preliminary  Qualification  Tests  (PQTs)  should  normally 
be  conducted  at  a  contractor's  development  facility  (i.e.,  contractor's 
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plant  or  other  location  of  available  equipment),  typically  using  con¬ 
trolled  inputs  specifically  prepared  for  the  test/demonstration  purpose. 
The  Test  Plan  should  outline  the  sequence  of  individual  and/or  assembled 
CPC  tests,  identify  the  CPCI  performance/design  requirements  to  be  veri¬ 
fied  at  each  PQT,  and  identify  special  simulation/recording  equipment 
or  other  support  requirements  for  the  PQT  program.  Each  Test  Procedures 
document  should  be  completed  by  the  contractor  and  submitted  to  the 
monitoring  agency  sufficiently  in  advance  of  the  scheduled  test  session 
(i.e.,  two  to  four  weeks)  to  permit  review  and  analysis  of  the  procedures 
prior  to  witnessing  the  testing  operations.  The  format  and  content  of  a 
typical  Test  Procedures  document  are  specified  in  DID  T- 103-1. 

(c)  Formal  Qualification  Testing  (PQT),  unlike  informal 
testing  and  PQT,  is  not  an  incremental  process.  FQT  is  designed  to  be 
an  integrated  and  comprehensive  functional  test  of  the  CPCI  as  a  whole, 
and  usually  is  conducted  in  one  continuous  time  period  Just  prior  to 
FCA.  Each  function  of  the  CPC'I  (as  delineated  in  the  CPCI  Part  I  Speci¬ 
fication)  is  tested  during  FQT,  regardless  of  the  amount  of  informal 
testing  and  PQT  conducted  previously.  Since  PQT  is  conducted  during 
the  on-going  design  process,  the  PQT  test  procedures  become  obsolete 
within  a  few  weeks  after  the  PQT  tests.  In  Contrast,  since  FQT  is  con¬ 
ducted  after  the  design  process  culminates  (just  prior  to  FCA),  the  FQT 
test  procedures  can  be  maintained  and  updated  and  used  throughout  the 
remainder  of  the  Full  Scale  Development  and  Deployment  Phases.  Thus, 
each  time  a  change  is  implemented  in  a  CPCI,  the  FQT  procedures  can  be 
used  as  a  test  tool  to  retest  the  CPCI  and  ensure  that  it  still  functions 
in  accordance  with  the  functional  requirements  set  forth  in  the  Part  I 
Specification . 


1.  For  the  less  complex  CPCIs,  or  for  those  (i.e., 
utility)  which  are  relatively  insensitive  to  the  system  operations, 
formal  qualification  will  usually  be  conducted  at  the  contractor's 
development  facility.  However,  for  a  large  operational  CPCI,  the  sheer 
complexity  of  the  performance  requirements  may  dictate  that  FQT  can  only 
be  conducted  in  the  context  of  the  operationally-configured  system,  in¬ 
cluding  personnel  and  communications .  Hence,  for  these  cases,  FQT  may 
often  not  be  conducted  at  the  contractor's  plant  (development  facility), 
but  may  require  use  of  the  System  test  site  prior  to  the  beginning  of 
System  testing. 


2.  In  the  event  that  a  complex  operational  CPCI  is 
scheduled  for  formal  qualification  at  the  System  test  site,  scheduling 
of  the  FQT  will  normally  be  prior  to  the  start  of  formal  System  test¬ 
ing,  but  following  a  period  of  contractor  adaptation,  installation,  and 
checkout  of  the  CPCI  in  the  previously  installed  and  tested  operational 
equipment/facilities . 

3-  In  either  case,  PQT  will  involve  the  use  of  sim¬ 
ulated  and  controlled  inputs  which  can  be  designed  to  cover  the  expected 
ranges  of  variables,  system  operational  modea  and  conditions,  including 
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capacities  and  limits .  Full  qualification  of  certain  functions  may 
depend  upon  subsequent  System  testing  with  live  inputs  and  communi¬ 
cations  . 


4.  The  procedures  for  Formal  Qualification  Tests 
(ref  DID  T-103-l)  will  be  in  the  some  format  as  the  PQT  procedures 
prepared  during  design  and  development.  However,  these  procedures  will 
emphasize  verifying  that  all  functional  requirements  are  met,  based  on 
the  demonstrated  performance  using  the  complete  CPCI.  Since  the  FQT 
procedures  require  approval  by  the  monitoring  agency,  they  should  be 
submitted  initially  in  preliminary  form.  Review  and  analysis  may  result 
in  proposed  changes  which  must  be  resolved  and  reflected  in  revised 
preparations  for  the  testing  activity,  tfence,  the  preliminary  document 
should  be  completed  in  sufficient  time  (i.e.,  three  months  prior  to 
scheduled  FQT)  for  this  potentially  lengthy  process  to  be  accomplished. 

3-  Levels  of  Cl/Suboystem  CPCI  Testing.  As  mentioned  earlier,  testing 
of  a  CPCI  is  required  at  three  levels:  subprogram  level,  functional 
level,  and  CPCI  level.  Regardles'3  of  the  type  of  testing  to  be  per¬ 
formed  (i.e.,  informal  versus  for.ial  testing),  these  three  levels  apply 
equally  to  each.  This  relationship  is  portrayed  in  Figure  3. 

a.  Subprogram  Testing.  Subprogram  testing,  sometimes  called  para¬ 
meter  testing,  is  directed  toward  ensuring  that  each  subprogram  (CFC  or 
lesser  entity)  interprets  its  inputs  correctly,  successfully  performs 
all  tasks  defined  in  the  subprogram  coding  specifications,  and  adheres 
to  prescribed  coding  conventions  and  standards.  Subprogram  testing  is 
the  most  detailed  and  basic  testing  that  is  performed  upon  computer 
programs.  Since  concentration  is  on  testing  basic,  easily  managed  units 
of  code,  the  detection,  isolation  and  correction  of  errors  that  are 
exceedingly  difficult  to  detect,  isolate  and  correct  in  later  phases  of 
testing  is  greatly  facilitated.  Therefore,  detailed  attention  must  be 
given  to  the  specification  of  rigid  test  requirements  so  that  errors 
are  not  overlooked  that  will  later  cause  unnecessary  testing  and  analysis . 
Each  subprogram  should  be  specifically  tested  for  arithmetic  and  logical 
accuracy  and  limitations,  usually  without  requiring  communication  with 
related  subprograms  or  external  equipment .  Subprogram  testing  commences 
following  the  development  of  a  set  of  coded  instructions  that  have  been 
subjected  to  detailed  visual  verification  (code  checking)  by  the  respon¬ 
sible  programmers,  and  automatic  code  analysis  by  the  applicable  compiler 
and/or  assembler  to  eliminate  errors  in  keypunching,  formatting  of  code, 
etc.  In  addition,  a  thorough  technical  review  of  the  subprogram  should 
be  conducted  by  contractor  personnel  to  ensure  that  violations  of  cod¬ 
ing  conventions  and  standards  have  been  corrected  prior  to  on-computer 
testing.  The  subpro  .rai  is  then  operated  with  a  range  of  data  that 
forces  use  of  all  decision  points  and  processing  paths .  The  test  re¬ 
sults  are  analyzed  to  determine  whether  the  derived  results  are  con¬ 
sistent  with  the  input  data,  the  results  expected  from  the  selected 
input  parameters,  and  the  operation  of  the  subprogram  as  defined  In  ood- 
ing  specifications. 


19 


20 


Figure  3.  Types  and  Levels  of  Computer  Program  Testing. 


b.  Functional  Area  Testing.  Functional  area  testing,  sometimes 
referred  to  as  string  or  assembly  testing,  is  directed  tovard  ensuring 
that  selected  sets  of  functionally  related  subprograms  interpret  all 
inputs  correctly,  successfully  perform  all  processing  tasks  specified 
v  in  performance  specifications,  and  generate  output  data  that  satisfies 

the  input  requirements  of  other  interfacing  functional  areas  or  equip¬ 
ments  .  Functional  area  testing  is  an  extension  of  subprogram  testing 
1  with  emphasis  being  placed  on  inter-program  communications  and  pro¬ 

cessing  of  data  within  a  defined  grouping  of  subprograms  or  components. 
Subprogram  testing,  on  the  other  hand,  concentrates  on  intra- program 
communications  and  data  processing.  Examples  of  functional  areas  are 
radar  inputs  processing  and  correlation,  data  link  inputs  processing, 
and  weapons  guidance.  Each  functional  area  is  composed  of  one  or  more 
subprograms  that  must  accept  as  inputs  other  functional  area  outputs, 
or  inputs  received  via  the  operating  hardware,  and  process  them  in 
accordance  with  functional  performance  specifications.  It  must  also 
prepare  data  for  use  by  other  sets  of  functionally  related  subprograms 
as  specified  in  detailed  coding  specifications.  In  some  cases,  (i.e., 
tape  read) ,  functional  area  testing  may  be  the  first  phase  of  testing 
performed.  This  will  most  often  occur  when  subprogram  testing  requires 
the  excessively  expensive  simulation  of  interfacing  hardware . 

(l)  Functional  area  testing  is  usually  the  first  point  at  which 
interactions  with  a  control  subprogram  are  examined.  Ideally,  the  ex¬ 
ecutive  or  control  function  is  the  first  to  enter  the  functional  area 
test  phase,  6ince  the  other  functions  can  be  more  conveniently  tested 
with  it.  In  actual  practice,  however,  the  control  functd-.m  may  not  be 
available  or  far  enough  along  in  development  for  this  purpose.  In  this 
event,  other  methods  for  control  may  be  utilized.  Although  many  func¬ 
tional  areas  are  usually  being  tested  at  the  same  time,  the  scope  of 
testing  gradually  increases  through  higher  and  higher  levels  of  assembly 
as  functiort;  become  verified  until  the  computer  program  is  ready  to 
undergo  CPC  I  testing.  In  practice  then,  functional  area  testing  begins 
as  "high-level"  subprogram  testing  and  phases  out  as  "low-level"  CPCI 
testing. 


(2)  Throughout  this  phase  of  testing,  the  performance  of  the 
i  computer  program  is  checked  against  that  specified  in  performance  re¬ 

quirements  decisions  and  limits  that  result  from  an  interface  between 
subprograms.  Therefore,  conditions  that  are  of  consequence  to  only  one 
-j  subprogram,  such  as  proper  exits  in  subroutine  decision  paths,  are 

better  tested  in  the  subprogram  test  phase  when  the  problems  of  defining 
and  establishing  the  proper  test  situations  are  not  so  complex.  In  ad¬ 
dition  to  differences  in  levels  of  concentration,  functional  area  test¬ 
ing  examines  interactions  between  all  functionally  related  components 
while  CPCI  testing  concentrates  on  interactions  between  functions.  Also, 
internally  stored  inputs  that  are  provided  to  the  subprograms  during 
the  functional  area  test  phase  usually  result  from  the  insertion  of  test 
data  by  a  test  input  tool.  CPCI  testing,  on  the  other  hand,  uses  Cl 
level  inputs  that  are  provided  either  by  other  CPCIs,  or  by  system  in¬ 
put  data  generators . 
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c.  CPC I  Testing.  CPCI  testing  is  directed  toward  ensuring  that  the 
total  package  of  computer  program  components  that  comprise  the  CPCI 
correctly  interprets  all  input  message  types  and  values,  accepting 
those  that  are  legal  and  properly  disposing  of  all  others;  properly 
processes  all  internally  stored  data;  and  correctly  formats,  arranges, 
and  outputs  all  required  system  data.  Thus,  CPCI  testing  verifies  that 
the  CPCs  operating  as  the  CPCI  fulfill  the  requirements  of  performance 
specifications .  Whereas  subprogram  testing  concentrates  on  verifying 
the  correct  operation  of  individual  computer  program  components ,  and 
functional  area  testing  emphasizes  inter-program  communications  within 
separable  functional  areas,  CPCI  testing  concentrates  on  verifying  that 
the  complete  set  of  computer  program  functional  areas  correctly  inter¬ 
acts  with  each  other.  Therefore,  CPCI  testing  must  he  accomplished 
after  functional  area  testing  and  prior  to,  or  in  parallel  with,  test¬ 
ing  with  other  CIs.  CPCI  testing  can  he  conducted  at  either  the  con¬ 
tractor's  in-plant  test  facility  or  an  installation  facility  (either 
operationally  employed  or  a  nonoperational  test  site).  During  the 
initial  stages  of  CPCI  testing,  the  bulk  of  the  testing  ia  conducted 
in-house  using  simulated  inputs.  This  provides  a  high  degree  of  con¬ 
fidence  prior  to  the  release  of  a  Cl  to  a  site  location.  During  the 
later  stages  of  CPCI  testing,  emphasis  shifts  from  the  testing  of  func¬ 
tional  conmunication  to  verification  of  overall  CPCI  performance  in  an 
operational  (or  very  nearly  so)  environment. 
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